A Virtual Director Inspired by Real Directors

Automatically computing a cinematographically consistent sequence of shots over a set of actions occurring in a 3D world is a complex task that requires not only the computation of appropriate shots (viewpoints) and appropriate transitions between shots (cuts), but more importantly the ability to encode and reproduce elements of cinematographic style that exist in real movies. In this paper, we propose an expressive automated cinematography model that learns some elements of style from real movies and reproduces them in synthetic movies. The model relies on a Hidden Markov Model representation of the editing process. The proposed model is more general than existing representations that encode cinematographic idioms and proves to be more expressive in the possible variations of style it offers. Introduction The recent possibilities in rendering increasingly realistic 3D virtual environments in real-time contexts urges the need for novel techniques to automatically and efficiently convey these contents through the use of appropriate cinematographic techniques. Indeed, there is a clear shift towards a more and more cinematographic experience of 3D environments especially in gaming contexts. More specifically, the reproduction of elements of cinematographic genre and style (war scenes, investigation scenes, etc.) plays a key role in the immersion of users. However most applications, including games, rely on manually preset camera viewpoints, pre-authored camera paths and triggered cuts between viewpoints. In attempts to replace this manual endeavor, the research community has proposed a number of automated approaches. The computation of automated cinematographic sequences either relies on search techniques in film-tree representations (Christianson et al. 1996) (a film-tree is the representation of a film in a hierarchical structure of scenes, shots, and frames), dynamic programming over collections of template shots (Elson and Riedl 2007), hierarchical planning (Jhala and Young 2005), or finite state machine representations (He, Cohen, and Salesin 1996) (in which a state Copyright c © 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. is a shot, and a transition is a cut between shots). While appraised for their generality, such representations display limitations when trying to perform variations in the way the sequence is shot and edited. Finite state machines encode filmic idioms (which are prototypical ways of shooting actions and actor blockings), and require the manual design of new idioms to perform variations. Film-tree techniques require the design of new heuristics to perform a different search in the film-tree. And dynamic programming techniques over collections of shots require the specification of different costs to perform variations in the generated results. Therefore, the task of mimicking elements of style from real movies is not straightforward. In this paper, we propose a new expressive model which can compute significant variations in terms of cinematographic style, with the ability to control the duration of shots, and adding specific constraints on the computed sequence. Furthermore, the model is parameterized in a way that facilitates the application of learning techniques so as to reproduce elements of style extracted from sequences of real movies. The approach is founded on a Hidden Markov Model representation of the editing process, where the states represent the shots and the observations are the events related to the shots. Related work A number of approaches have been proposed in the literature to partially or fully automate the computation of viewpoints and edits. The seminal work of He etal. (He, Cohen, and Salesin 1996) proposes to encode the directorial process (placing the cameras and selecting the cuts between the cameras) as a finite state machine (FSM). A state in the FSM represents a canonical shot (eg. apex shot, overthe-shoulder shot or panoramic), while a transition between states represents a cut. The transitions are either triggered by scene events (eg. when the distance between two characters reaches a threshold) or temporal events (duration of the current shot). The FSM representation is then used in a real-time context to compute a cinematographic sequence: as the 3D scene evolves, the FSM places the camera corresponding to the shot expressed in the current state, and performs the cut when events occur. By relying on Arijon’s cookbook (Arijon 1976) of camera setups, the authors encode different filmic idioms (stereotypical ways of shooting Intelligent Cinematography and Editing: Papers from the AAAI-14 Workshop